Skip to content

BE-457: HashQL: MIR execution pipeline extensions for postgres compilation#8525

Open
indietyp wants to merge 4 commits into
mainfrom
bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres
Open

BE-457: HashQL: MIR execution pipeline extensions for postgres compilation#8525
indietyp wants to merge 4 commits into
mainfrom
bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres

Conversation

@indietyp
Copy link
Copy Markdown
Member

@indietyp indietyp commented Mar 8, 2026

🌟 What is the purpose of this PR?

Prepares the MIR execution pipeline for consumption by the postgres compiler. The execution analysis now produces a complete IslandGraph (not just a flat island list), the placement solver and island placement accept external allocators, and the traversal system gains the TraversalPathBitMap and as_symbol() APIs that the SQL generator needs. Also adds a backend switch cost to the terminator placement so cross-backend transitions are no longer free.

🔍 What does this change?

Execution analysis (pass/execution/mod.rs):

  • run() becomes run_in() with an explicit allocator parameter, returning ExecutionAnalysisResidual (assignment + island graph). The island graph is now constructed as part of the analysis rather than left to the caller.
  • Adds run_all_in() which runs the execution analysis over all graph-read bodies in a DefIdSlice.

Traversal system (traversal/mod.rs, traversal/entity.rs):

  • TraversalPathBitMap: per-vertex-type collection of TraversalPathBitSets with pointwise lattice operations. The postgres compiler uses this to track which paths each island accesses across all vertex types.
  • TraversalPath::as_symbol(): returns a static symbol for each path variant, used as SQL column aliases so the interpreter can locate result columns by name.
  • TraversalPathBitSet::vertex(): returns the vertex type for a bitset.
  • EntityPath::as_symbol() and EntityPath::column_name(): per-path SQL identifiers.

Terminator placement (terminator_placement/mod.rs):

  • TransMatrix gains AddAssign for element-wise saturating addition.
  • Adds backend_switch_cost() which encodes a fixed overhead for cross-backend transitions (Postgres to Interpreter: 8, Interpreter to Embedding: 4, etc.). Previously cross-backend transitions had zero inherent cost, so empty blocks were arbitrarily assigned to the interpreter even when staying on postgres was free.

Placement solver (placement/solve/):

  • PlacementSolver::run() becomes run_in() with allocator parameter.
  • CSP solver internal allocations use the provided allocator.

Island graph (island/graph/mod.rs):

  • IslandGraph::new_in() now takes an allocator for its output storage.

Pretty printer (pretty/text.rs):

  • TextFormatAnnotations gains annotate_basic_block() and BasicBlockAnnotation associated type, plus a blanket impl for &mut T.

Builder (builder/rvalue.rs):

  • Adds RValueBuilder::opaque_entity_uuid() convenience constructor for the common pattern of extracting an entity's UUID.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

⚠️ Known issues

The backend_switch_cost() values (8, 4, etc.) are hand-tuned heuristics. They correctly prevent the solver from arbitrarily switching backends for empty blocks, but a proper cost model would derive these from measured overhead.

🛡 What tests cover this?

  • Updated execution pass end-to-end tests (execution/tests.rs) including entity_uuid_equality, mixed_postgres_embedding_interpreter, projection_and_apply_splits
  • Updated statement placement tests for interpret and postgres backends (eq_opaque_entity_uuid)
  • Updated terminator placement and island tests
  • Placement solver tests updated for allocator API

❓ How to test this?

cargo nextest run -p hashql-mir

feat: checkpoint (II)

feat: checkpoint (III)

feat: snapshot vec

feat: add dedicated filter

feat: checkpoint

feat: filter implementation

feat: filter implementation (mostly) done

chore: environment capture note

chore: always postgres bigint

feat: target clone

feat: simplify lookup

feat: move storage up

feat: eval entity path

chore: checkpoint

chore: checkpoint

chore: find entrypoint

feat: eval context

feat: eval cleanup

chore: cleanup

feat: track index

feat: wire up filter

feat: add error reporting

chore: checkpoint

feat: add traverse, and first postgres compiler outline

feat: traverse bitmap

feat: move traversal out

feat: projections

feat: projections

fix: clippy

feat: subquery projection for lateral

feat: checkpoint

feat: test plan

feat: checkpoint

feat: checkpoint – failing tests ;-;

feat: checkpoint – failing tests ;-;

feat: checkpoint — passing tests

fix: import

fix: entity type

feat: checkpoint

feat: attribute a cost to terminator placement switches

fix: import

feat: checkpoint

feat: checkpoint

chore: lint
@cursor
Copy link
Copy Markdown

cursor Bot commented Mar 8, 2026

PR Summary

Medium Risk
Medium risk because it refactors allocator plumbing and core execution/placement analysis APIs, and changes backend transition costing which can alter target assignments and scheduling behavior across queries.

Overview
Refactors ExecutionAnalysis to be allocator-parameterized (run_in) and to return a new ExecutionAnalysisResidual that includes both the per-block target assignment and a fully-built IslandGraph; adds run_all_in to run this over all GraphReadFilter bodies.

Upgrades island/traversal plumbing for SQL generation: IslandGraph gains block membership helpers and find, traversal paths gain stable as_symbol() identifiers plus a new per-vertex TraversalPathBitMap lattice, and VertexType becomes an Id (ordered/repr(u8)) for indexed maps.

Changes placement behavior by introducing a fixed backend switch overhead in terminator transition costs (via TransMatrix AddAssign) so cross-backend edges are no longer effectively free; updates the placement solver and island placement APIs to accept external allocators, adds MIR builder support for opaque(...) rvalues, and refreshes benches/snapshots/tests (including new EntityUuid opaque equality coverage and pretty-printer basic-block annotations).

Reviewed by Cursor Bugbot for commit 39f364e. Bugbot is set up for automated code reviews on this repo. Configure here.

@vercel
Copy link
Copy Markdown

vercel Bot commented Mar 8, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview, Comment May 12, 2026 2:09pm
petrinaut Ready Ready Preview May 12, 2026 2:09pm
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
hashdotdesign Ignored Ignored Preview May 12, 2026 2:09pm
hashdotdesign-tokens Ignored Ignored Preview May 12, 2026 2:09pm

Copy link
Copy Markdown
Member Author

indietyp commented Mar 8, 2026

This stack of pull requests is managed by Graphite. Learn more about stacking.

@augmentcode
Copy link
Copy Markdown

augmentcode Bot commented Mar 8, 2026

🤖 Augment PR Summary

Summary: Prepares the HashQL MIR execution pipeline for Postgres compilation by returning a full island dependency graph and exposing traversal/placement APIs needed by the SQL generator.
Key Changes:

  • Execution analysis now takes an explicit allocator (run_in) and returns ExecutionAnalysisResidual (assignment + IslandGraph); adds run_all_in for batch analysis.
  • Traversal adds TraversalPathBitMap, TraversalPath::as_symbol(), and EntityPath::as_symbol()/column_name() for stable SQL column aliases.
  • Placement solver and island placement APIs accept external allocators and route CSP internal allocations through the provided bump allocator.
  • Terminator placement introduces a fixed backend_switch_cost() so cross-backend transitions have non-zero overhead (avoids “free” backend flips).
  • Island graph gains helper queries (members, contains, find) and is now built/resolved within execution analysis.
  • Pretty printer supports per-basic-block annotations; MIR builder gains an opaque aggregate constructor and new tests cover opaque UUID equality.
Tests: Updates/extends execution, statement placement, terminator placement, and allocator-related solver snapshots (incl. entity_uuid_equality). Notes: Backend switch costs are heuristics intended to be refined by a measured cost model.

🤖 Was this summary useful? React with 👍 or 👎

Copy link
Copy Markdown

@augmentcode augmentcode Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. No suggestions at this time.

Comment augment review to trigger a new review at any time.

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 8, 2026

Codecov Report

❌ Patch coverage is 64.24581% with 128 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.31%. Comparing base (a18ae88) to head (39f364e).

Files with missing lines Patch % Lines
...cal/hashql/mir/src/pass/execution/traversal/mod.rs 0.00% 44 Missing ⚠️
libs/@local/hashql/mir/src/pretty/text.rs 21.62% 28 Missing and 1 partial ⚠️
.../hashql/mir/src/pass/execution/traversal/entity.rs 0.00% 28 Missing ⚠️
libs/@local/hashql/mir/src/pass/execution/mod.rs 44.00% 14 Missing ⚠️
.../hashql/mir/src/pass/execution/island/graph/mod.rs 35.29% 11 Missing ⚠️
.../mir/src/pass/execution/placement/solve/csp/mod.rs 90.00% 0 Missing and 1 partial ⚠️
libs/@local/hashql/mir/src/pass/execution/tests.rs 99.07% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main    #8525    +/-   ##
========================================
  Coverage   62.31%   62.31%            
========================================
  Files        1354     1354            
  Lines      137003   137313   +310     
  Branches     5792     5793     +1     
========================================
+ Hits        85372    85567   +195     
- Misses      50725    50839   +114     
- Partials      906      907     +1     
Flag Coverage Δ
rust.hashql-compiletest 29.63% <ø> (ø)
rust.hashql-mir 92.08% <64.24%> (-0.37%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Mar 8, 2026

Merging this PR will not alter performance

✅ 24 untouched benchmarks
⏩ 56 skipped benchmarks1


Comparing bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres (39f364e) with main (a18ae88)

Open in CodSpeed

Footnotes

  1. 56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 6ef6d13 to ef9b858 Compare March 8, 2026 16:15
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 5d95ba7 to 1aa0f1c Compare March 8, 2026 16:15
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from ef9b858 to a91293c Compare March 8, 2026 16:56
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 1aa0f1c to 87024df Compare March 8, 2026 16:56
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 5552cbb. Configure here.

Comment thread libs/@local/hashql/mir/src/pass/execution/mod.rs
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 5552cbb to 60da845 Compare May 4, 2026 11:34
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 60da845 to fd9a3a8 Compare May 8, 2026 08:33
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 4f33a35 to 959fba1 Compare May 11, 2026 10:58
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from fd9a3a8 to 44ba0c2 Compare May 11, 2026 10:58
Base automatically changed from bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query to main May 12, 2026 06:38
@indietyp indietyp requested a review from a team as a code owner May 12, 2026 06:38
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 44ba0c2 to 39f364e Compare May 12, 2026 13:58
Copy link
Copy Markdown
Contributor

@thehabbos007 thehabbos007 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good nothing stood out to me that needed to change

}

impl<'alloc, A: Allocator, S: BumpAllocator> PlacementSolver<'_, 'alloc, A, S> {
impl<'alloc, S1: Allocator, S: BumpAllocator> PlacementSolver<'_, 'alloc, S1, S> {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small nit: I suppose it is slighly confusing to me that we name the allocator S1 and bump allocator S. Is there a reason why this naming is used instead of A/B or other separate letters? I suppose S' would have been slightly nicer but S1 does fill that wish :)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would've called it S' (but we live in a world where that isn't a thing in Rust :P) I work through this upstream, in general there should only ever be two allocators present: A, the allocator that we deposit into, and S, the scratch that isn't long-term persistent. That's it (hence the naming here). I believe that I reconcile this properly later. The issue becomes one we have the A inside of the function, instead of the trait because then A can be anything, and we create three different generics instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

3 participants